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Apparatus for weighing, sizing and defect sorting of fruit comprising one or more singulators (1 1) to form the fruit into one 
or more rows, a conveyor that is arranged to pass the rows of fruit underneath a charged couple device array camera (19), means 
(98, 99) to rotate each piece of fruit (100) as it passes underneath the camera (19), an image processing means for processing a 
captured image, the image processing means comprises a master processor (102) which passes signals to a vision processor (103) 
and onto at least eight object processors (104-1 11) which divide the captured image into image sections with each image section 
containing a representation of only one of the pieces of fruit. The object processors (104-111) simultaneously process each image 
section and communicate information on the processed sections to control means. The control means includes means to classify 
the objects according to predetermined criteria using the analysed output. The predetermined criteria can relate to size, weight 
and especially colour. The apparatus includes ejection means (50) spaced along the length of the conveyor controlled by the con- 
trol means to eject the fruit to outfeed conveyors (15) at particular locations corresponding to the predetermined criteria. 
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Title 

CIASSIFYING AND SORTING OF OBJECTS 
Field of Invenl^ion 

This invention relates to the classifying and 
sorting of objects and includes a sizing, colouring, and 
defect sorting conveyor system for use with a wide variety 
of both primary produce and packaged and unpackaged 
foodstuffs. In particular, the invention relates to a 
conveyor system that is used for the colour and blemish 
sorting and dimensional sizing of various fresh fruits . 
Prior Art 

Over the years there have been many proposals 
associated with the sizing and weighing of foodstuffs, 
especially fresh fruit. There are a number of 
sophisticated conveyor systems that weigh and size fruit 
that is passed along the conveyor system to automatically 
eject the fruit at predetermined locations so that the 
fruit can be sorted and collected in accordance with size 
and weight . There have also been proposals to incorporate 
photographic means to take pictures of the fruit as they 
pass along the conveyor in an attempt to colour and blemish 
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sort the fruit. However, these attempts to colour and 
blemish sort fruit have been only partially successful and 
it is considered that there is a need for a far more 
sophisticated and accurate colour and blemish sorter that 
can be used not only with fruit, but many other types of 
primary produce and both packaged and unpackaged 
foodstuffs • 

It is this need that has brought about the present 
invention . 

Summary of the inviention 

According to the present invention there is 
provided apparatus for classifying and sorting objects 
comprising: 

a singulator to form the objects into a least one 

row; 

a conveyor that is arranged to pass at least one 
row of the objects underneath a caonera, the camera being 
capable of capturing at least one image of the objects; 

means t.o rotate each object whilst passing 
\indemeath the camera; 

an image processing means for processing of the 
captured image, the image processing means including means 
for dividing the captured image into image sections, each 
image section containing a representation of only one of 
the objects, and the image processing means in use 
performing further processing on each of the image sections 
simultaneously; and 

a control means for analysing output from the image 
processing means, classifying the objects according to 
predetermined criteria using the analysed output, and 
controlling ejection means to reject the objects at 
particular locations on the conveyor corresponding to the 
predetermined criteria. 
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According to the present invention there is further 
provided a method of classifying and sorting objects 
comprising the steps of : 

singulating the objects into at least one row; 

passing at least one row of the objects on a 
conveyor past a Cctmera; 

rotating each object whilst passing underneath the 

camera ; 

capturing at least one image of each object; 
digitising the captured image; 

dividing the captured image into a plurality of 
image sections, each image section containing a 
representation of only one object; 

analysing the plurality of image sections 
simultaneously to determine information on the colour, size 
and/or shape of each object; ^^-^ 

classifying the objects by comparing the: determined 
information of each object with predetermined criteria; and 

selectively ejecting the objects from the conveyor 
at stations corresponding to the predetermined criteria . 
Description of the Drawings 

Embodiments of the present invention will now be 
described, by way of example only, with reference to the 
accompanying drawings in which: 

Figure 1 is a plan view of apparatus for size and 
blemish sorting fruit. 

Figure 2 is a side elevational view of the 
apparatus in Figure 1, 

Figure 3 is a cross-sectional view taken along the 
lines 3-3 of Figure 1 illustrating a photographic section. 

Figure 4 is an illustration of a view that would be 
seen by a camera in the photographic section. 

Figure 5 is a cross-sectional view taken along the 
lines 5-5 of Figure 1, 

Figure 6 is a cross-sectional view taken along the 
lines 6-6 of Figure 5, 
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Figure 7 is a perspective view of a roller shaft 
chain clip forming part of the conveyor. 

Figure 8 is a perspective view of an ejector lever 
chain clip and ejector lever forming part of the conveyor. 

Figure 9 is a sectional view taken along the lines 
9-9 of Figure 1 illustrating an ejection station. 

Figure 10 is a cross-sectional view taken along 
the lines 10-10 of Figure 9, 

Figure 11 is a cross -sectional view similar to 
Figure 1 but illustrating a second embodiment of the 
invention. 

Figure 12 is a block diagram illustrating the 
computer of the classification system connected to a single 
camera , 

Figure 13 illustrates the functional equivalent of 
Figure 12 and the Ccunera/conveyor synchronization process. 

Figure 14 shows a functional diagram of a two 
camera system. 

Figure 15 illustrates the hxman eye's sensitivity 
to colour , 

Figure 16 illustrates the RGB representation of a 
coloured point. 

Figure 17 shows slices of image sections used by 
the object processors. 

Figure 18 is a graphical representation of the 
two-dimensional classification map. 

Figure 19 shows a classification criteria 
boundaries on the classification map. 

Figure 20 illustrates the object volume estimation 
technique, and 

Figure 21 illustrates the object diameter 
estimation technique. 
Description of the Embodiments 

As shown in Figures 1 and 2 a conveyor system for 
fresh fruit comprises essentially a feed conveyor 10 that 
feeds the fruit to a singulator 11 that positions the fruit 
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individually in two rows, each piece of fruit being 
supported centrally between a pair of rollers. The two 
parallel rows of separated fruit pass under a photographic 
section 12 to an ejection section 13 which includes a 
plurality of spaced ejector positions on either side of the 
asseinbly that have the effect of causing particular fruit 
to be ejected onto a plurality of outfeed conveyors 15 that 
extend substantially transversely to the direction of 
movement of the main conveyor. The conveyor system is 
supported about a metallic framework 29 to be freestanding 

as shown in Figure. 1. 

The conveyor system operates so that the fruit are 
fed from the feed conveyor 10 onto the singulator 11 which 
causes the fruit to pass in two separate rows through the 
photographic section 12. Whilst in the photographic 
section 12, the fruit are rotated and various photographs 
are taken by a charged couple device (CCD) array camera 19 
to provide a full colour video signal corresponding to each 
photograph. A computer comprising a series of 
microprocessors makes quality and quantity decisions about 
the fruit that are photographed by the camera. A master 
processor unit comprising a system of master processor and 
camera processor communicates with a vision processor and a 
plurality of object processors to classify the fruit 
according to predetermined sizing, colour and defect 
criteria. As the fruit pass through the photographic 
section 12, and into the ejection section 13, the computer 
periodically signals ejector assemblies to eject certain 
fruit at corresponding ejector positions on the conveyor in 
accordance with the classification criteria used. These 
fruit are fed into outfeed conveyors 15 which then 
transport the fruit to suitable storage means . In the 
first part of this description, a detailed explanation is 
given of the mechanical componentary that makes up the 
conveyor system and in the second part of this description. 
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there is reference to the theory and technology that 
underlies the fruit classification means that utilises the 
camera signal to achieve the end result of sorting and 
ejecting ceirtain fruit from the conveyor. 
Mechanical Details 

As shown in Figures 1 and 2, the conveyor includes 
an infeed conveyor 10 which is of conventional form that 
supplies the fruit to the conveyor system. The infeed 
conveyor is arranged to feed fruit to the singulator 11 
which comprises a horizontally positioned conveying surface 
110 having inclined singulator conveyors 111 and 112 
positioned on either side. The inclined conveyors 111 and 
112 operate at half the speed of the hoirizontal conveyor 
110 and have a series of spaced plates extending across the 
conveying surfaces. The inclined conveyors 111 and 112 
cooperate with the horizontal conveyor 110 to, in effect 
juggle the fruit to ensure that they are formed into a 
single row so that they can feed in single file to the main 
convejror system. Although not shown in the drawings, the 
inclined singulator conveyors 111 and 112 are arranged to 
extend past the end of the horizontal singulator conveyor 

110 alongside the start of the main conveyor. In this 
manner, in the event that more than one piece of fruit 
attempts to sit in the gaps between roller assemblies that 
form part of the main conveyor, the fruit is joggled side- 
ways against the slowly moving singulator conveyor belts 

111 and 112 and is held there until a space appears between 
two rollers thereby ensuring that complete singulation 
takes place. The inclination of the singulator conveyors 
111 and 112 is approximately 30 degrees to the horizontal. 

The main conveyor system which includes the 
singulator 11, photographic section 12 and ejection section 
13 comprises a pair of spaced apart continuous chains 20 
each driven by a pair of sprockets 21, 22 at either end. 
It is understood that the drive to the sprockets 21, 22 is 
undertaken -by conventional means . The pair of continuous 
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chains are shown in cross section in Figure 5 where it can 
be seen that each chain 20 runs on an elongate plastics 
runner 25 that is secured to extend longitudinally of an 
extzraded aluminium section 26. The aluminium section has 
upstanding arm portions 21 , 28 that fit and locate the 
runners in two spaced apart parallel horizontal arrays. 
The extruded alximinixim section 26 extends along the 
underside of the upper run of the conveyor. The chains 20 
are used to drive a plurality of parallel extending spaced 
apart roller assemblies 30 each of which extends 
transversely of the chains and comprises a transverse drive 
shaft 31 that coaxially supports a plastics roller 32 with 
a concavely tapered exterior surface 33. The rollers 32 
are mounted on the drive shaft 31 so that rotation of the 
drive shaft 31 can impart rotation to the rollers. The 
drive shaft 31 is attached to the chains 20 through a-« 
plurality of roller shaft chain clips 40, shown in cross 
section in Figure 5 and perspectively in Figure 7. Each 
roller shaft chain clip 40 comprises a plastics moulding 
with a central rectangular shaped box section 41 having a 
pair of coaxially aligned apertures 42 through which the 
drive shaft 31 freely extends. The rectangular box section 
41 terminates in a pair of downwardly extending legs 44 
which terminate in inwardly facing clip portions 45 which 
are arranged to clip under the link 29 of the chain 20 as 
shown in Figures 5 and 7. The chain clips 40 allow the 
drive shafts 31 to be clipped across the chains 20 so that 
movement of the chains by the spigot drive causes the 
roller assemblies 30 to move along the conveyor. Each 
roller 32 with its concave outer surface 33 is arranged so 
that each fruit 100 on entering the conveyor from the feed 
conveyor 11 is able to roll into a freestanding position 
where it is wedged between adjacent rollers^ see Figures 2 
and 3. The spacing of the rollers and their concave outer 
surfaces allow fruit of various sizes to be supported 
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between adjacent rollers. As the fruit passes up the 
singulation section 11 of the conveyor, the rollers 32 
shuffle the fruit into position so that they enter the 
photographic section 12 supported in equilibrium between 
adjacent rollers 32. 

As shown in Figure 3^ the photographic section 12 
comprises an enclosure 95 that is positioned with its open 
base 61 closely above the conveyor, the enclosure 95 has 
its interior side walls 62 with a suitable reflective 
material. A pair of electric lights 63 are positioned 
adjacent the upper end of the rectangular enclosure with 
the light pointing towards the comers of the enclosure 95 
as shown in Figure 2. The CCD video camera 19 is 
positioned centrally of the enclosure with the lens 64 
projecting downwardly to point at the two rows of fruit 
that are passing through the photographic section. In 
Figure 4, there is an illustration of what the camera sees 
as it takes a picture of the two rows of fruit as they pass 
through the enclosure. The picture comprises eight 
rectangles that overlap at adjacent edges so that some of 
the rectangles embrace a single piece of fruit and the 
edges of the adjacent fruit. The images in the rectangles 
are transferred to eight object processors for 
inteirpretation as described later in the specification. 

As shown in Figure 5 a circular wheel 98 is 
attached to the mid point of each roller support shaft 31. 
When the support shafts pass into the photographic section 
12, the wheels 98 are arranged to ride against an abutment 
99 that extends upwardly from the aluminium section 26. 
When the wheel 98 engages the abutment 99, the longitudinal 
motion of the conveyor causes the wheel 98 to rotate to 
drive the shaft 31 to in turn drive the rollers 33. This 
has the effect of causing the fruit located between 
adjacent rollers, to rotate through 360 degrees as the 
fruit passes through the photographic section. Rotation of 
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the fruit: ensures that all sides of the fruit are exposed 
to the camera whilst four photographic shots are taken. 

Once the fruit has passed photographic section 12, 
the fruit reaches the ejection section 13 which comprises a 
plurality of longitudinally spaced ejector assemblies 50 
shown in Figures 1 and 2. The ejector assemblies 50 are 
coupled to outfeed conveyors 15 that extend transversely of 
the conveyor system to deposit the fruit into respective 
bins or other collection means (not shown) . The outfeed 
conveyors 15 are of conventional design and are not 
described in detail in this specification. 

Each ejector assembly 50 is illustrated with 
particular reference to Figures 5, 6, 9 and 10. At 
periodic positions along the ejection section 13 of the 
conveyor^ an ejector assembly 50 is provided to eject fruit 
from the roller assemblies 30. At these positions as«^ shown 
in Figures 6 and 8, an ejector chain clip 60 is used to 
hold the ejector assemblies 50 onto the chains 20. The 
ejector chain clip 60 is illustrated in detail in Figure 8 
and comprises a plastics moulding that has a pair of 
downwardly extending parallel legs 61 that terminate at 
their free ends in outwardly extending clip portions 6 2 
that clip against the inside of the chain links 29 as shown 
in Figure 8. The legs 61 are attached at their upper ends 
to a pair of parallel spaced apart flanges 64 which extend 
transversely to one side of the legs 61. The ends 65 of 
the flanges opposite the legs towards its upper portion, 
are provided with a pair of coaxial apertures 66 ^ which are 
arranged to support pins 67 that extend from either side of 
an ejector lever 68. The ejector clip 60 is clipped onto 
the chain 20 so that the parallel side flanges 64 extend 
inwardly of the conveyor. The ejector lever 68 is provided 
with an elongate ejector arm 68 that is inclined to a 
median portion 70 through an angle at about 135 degrees. 
The median portion 70 merges into a downwardly extending 
abutment arm 71 that extends at an angle of about 135 
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degrees to the median portion • As shown in Figure 5^ the 
lever 68 is positioned so that the ejector arm 69 extends 
dov/nwardly between the sides of two adjacent rollers 32 
with the abutment arm 71 also extending downwardly of the 
pivot point defined by the pins 67. The ejector lever is 
arranged to pivot about the pivot point with the ejector 
arm 69 moving upwardly to the full line profile shown in 
Figure 8. This has the effect of forcing the fruit 100 on 
the roller 32 to roll outwardly of the conveyor, and as is 
shown in Figures 9 and 10 against a large rotating 
cylindrical brush SO that encourages the fruit to move onto 
the outf eed conveyor 15 which in turn transfers the fruit 
to suitable collection means . As shown in Figures 9 and 
10/ the brushes 80 are driven through chains 151, 152 and 
gearing 153 to ensure that the fruit have a soft landing 
when they are ejected from the main conveyor onto the 
outf eed conveyor. The brushes 80 rotate clockwise as shown 
in Figure 9 . 

A solenoid assembly 90 that is illustrated in 
Figures 5 and 6 comprises a rotary solenoid 91 that is 
mounted on a mounting plate 92 that forms part of the 
aluminium section 26 that extends across the conveyor. The 
solenoid has a trip arm 93 of triangular cross section when 
viewed in plan that sits on the top of the rotary solenoid 
91 and rotation of the trip arm in the horizontal plane 
(Figure 6) has the effect of forcing the abutment arm 71 of 
the ejector lever 68 to the dotted profile shown in Figures 
5 and 6, this has the effect, as described above of 
ejecting fruit from the conveyor. The rotary solenoid 
assembly is activated by an electrical signal that comes 
from the computer. 

One of the problems that can be encountered with 
conveyors of the kind described above is that due to the 
high speed of operation, i.e., fruit travelling at 
approximately 10 fruit per second and the comparatively 
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light-weight component ary, there is a danger that if the 
ejector levers 68 do not return to the neutral position 
shown in dotted profile in Figure 5, the abutment arm 71 of 
each ejector lever 68 could, as it moves rapidly past the 
fixed componentary of the conveyor such as the solenoid arm 
93 have a direct collison against one of the solenoid arms 
which could resui.t in disintegration of the plastics 
componentary the effect of which would be to destroy a 
substantial length of the whole assembly* In appreciation 
of malfunctions such as a solenoid breakdown which could 
result in the sort of collison referred to above, a safety 
ramp assembly 150 is motmted on an upstanding web 151 that 
is secured to the aluminitim extrusion 26 as shown in 
Figures 5 and 6. The safety ramp assembly 150 incorporates 
an inclined piece of tubing 153 that is inclined to the 
vertical as shown in Figure 5 at an angle of 45 degrees and 
to the horizontal through an angle of about 15 degrees as 
shown in Figure 6. The inclined piece of tubing 153 acts 
as a ramp so that in the event of the abutment arm 71 
directly contacting the tube due to failing to return to 
the fully inoperative position shown in dotted profile in 
Figure 5, the ramp has the effect of forcing the arm 71 
either to the left or the right of the ramp, i.e., either 
to the full line or dotted line positions of Figure 5 
thereby avoiding the ensuing damage of a collison. Thus, 
the safety ramp assembly 150 might have the effect of 
causing one or more errors in firuit ejection but these 
errors are considered very slight compared with the 
consequences of a solenoid failure causing a direct 
collison with componentary of the conveyor system. 
Although not shown in detail in Figures 5 and 6 the 
actuator arm 93 of the solenoid is designed to complete its 
movement from the inoperative to operative positions shown 
in Figure 6 without impeding the safety ramp assembly 150. 
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In the embodiment shown in Figures 1 to 10, a 
balanced assembly is defined whereby rollers 32 are 
positioned on the either end of a transversely extending 
drive shaft 31 that is driven via a drive wheel 98 running 
on a friction bed 99. The disadvantage with this assembly 
is that it is comparatively bulky. The embodiment shown in 
Figure 11 comprises a streamlined assembly whereby the 
rollers are axially split to define two a pair of 
sub-rollers 250 and 251 interconnected by a short stub 
axial 253 that is in turn clipped onto the chain link 29 in 
a manner substantially the same as in the first embodiment. 
An ejector assembly 50 which is also substantially similar 
to the first embodiment is clipped between adjacent roller 
sub-sets as viewed in plan and each sub-set of rollers is 
rotated by contact of the lowermost surface 255 of the 
chain clip on to a centrally positioned running surface 256 
that extends along the length of the conveyor. The ejector 
assembly 50 operates in a slightly different manner to that 
described in the earlier embodiment in that the abutment 
arms 271 are arranged to be coupled to either side of the 
clip 260 and in the inoperative position assume the full 
line profile with the abutment arm 271 extending vertically 
downwardly. A solenoid mechanism 290 similar to that of 
the earlier embodiment is operable to cause the arm 271 to 
be forced inwardly to the dotted profile shown in Figure 11 
whereby the other end of the ejector assembly 50 moves 
upwardly to eject the fruit laterally of the conveyor. The 
assembly also includes a safety ramp device 250 
substantially similar to that of the first embodiment and 
the arm 293 on the solenoid 290 is arranged to act behind 
the abutment arm 271 to, in use, cause the inward motion of 
the abutment arm from the inoperative to the operative 
positions. This assembly has a distinct advantage that it 
takes up considerably less space than the assembly 
described in the first embodiment. 
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The outfeed conveyors 15 and their associated 
ejector assemblies are arranged so that fruit of particular 
sizes, colours and defects can be ejected at particular 
points along the ejection section 13 of the conveyor. The 
ejection is initiated by signals that come from the 
computer that has interpreted the signals from the video 
camera r compared these signals with predetermined input 
data to determine the classification of sizing, colour and 
defect and then produces periodic electrical signals to the 
rotary solenoids to effect ejection at the required 
positions • 

The Classification System 

The mechanical components of the conveyor system, 
as described above, enable fruit to be manipulated and 
sorted into classes of similar fruits. The following 
description provides a more detailed outline of how the 
fruit are classified into their appropriate classes. 

The classification system is comprised of a 
computer which is connected to at least one camera. See 
Figure 12- The camera is positioned at a point along the 
conveyor system, as described above- The system described 
below employs only one camera, but more than one camera may 
be employed in order to obtain more comprehensive pictures 
of each piece of fruit, or to handle larger numbers of 
fruit at the same time. 

The system is based upon the analysis of a full 
colour video signal from a Charged Couple Device (CCD) 
array camera 101 . The computer includes a maister 
processing unit which is comprised of a master processor 
(not shown) and a camera master processor 102. The master 
processor is in overall control of the classification 
system, whilst the camera master processor is in control of 
the camera(s). The camera master processor passes on the 
video signals to the vision processor 103 for preliminary 
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image processing. Processed image data is then passed 
along the colour bus 113 to up to eight individual object 
processors for further processing. The object processors 
then pass information regarding the processed images to the 
master processor via the RTI-1000 bus 112. This 
arrangement is illustrated in Figure 12. 

Figure 13 shovms a two camera system. The two 
cameras are synchronised and produce images which provide 
greater coverage of the whole surface area of a piece of 
fruit. Alternatively, the second camera can photograph a 
second piece of fruit in order to increase the rate of 
fruit classification. 

As shown in Figure 3, the camera 19 is positioned 
above the conveyor system 32. The fruit 100 are 
illuminated by the light sources 63. A number of pieces of 
fruit move into the field of view 71 of the camera at one 
time. In the present system, two rows of fruit enter the 
field of view of the camera and each row may contain up to 
four pieces of fruit. Consequently, up to eight pieces of 
fruit may be in the field of view of the camera. 

The camera takes the first available photograph 
after the new cup sync (synchronisation) signal is 
received. The vision processor in the computer grabs the 
picture and digitises this picture. Analysis and 
classification of the fruit is performed using this 
digitised image. At this stage, up to eight pieces of 
fruit may be represented in the one image. Consequently, 
it is necessary to divide the image into eight image 
sections , each section containing the image of only one 
piece of fruit. Figure 4 illustrates the whole picture 
taken by the camera, and shows the eight individual image 
sections corresponding to each piece of frxiit. Each of the 
image sections is sent via the coloxir bus to a 
corresponding object processor. Consequently, eight 
individual object processors can simultaneously process 
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images of eight different pieces of fruit • This form of 
parallel processing significantly increases the rate of 
classification of the system. Each object processor 
further processes the image; this will be described further 
below. 

Once an image has been captured by the camera / 
digitised, analysed and divided by the vision processor and 
sent to the corresponding object processors for further 
processing, the fruit would have moved along the conveyor 
system and would have rotated by a certain amount . Two 
pieces of fruit would have moved out of the view of the 
camera while two new pieces of fruit would have moved into, 
the view of the cameras. A new picture will be taken and 
processed, as described above. This process continues so 
that up to four individual pictures of each piece of fruit 
are obtained. Each view should provide a different dLmage 
of a different part of the surface area of the piece of 
fruit. 

The four different views of the fruit are sent to 
an object processor. Similar views of different fruit are 
sent to other object processors, thus resulting in up to 
eight pieces of fruit being photographed and analysed at 
the same time. Once detail of the colour, grade and size 
of each piece of fruit is extracted by the object processor 
from the four views , the information is sent to the master 
processor by each object processor via the RTI-1000 bus. 

An operator is able to specify the classification 
criteria to be used in sorting the fruit. For example, it 
may be required to sort the fruit according to colour into 
six different groups. Information regarding the range of 
colours of fruit to be sorted into each group is entered 
into the master processor by the operator. Upon receiving 
information on a piece of fruit from an object processor, 
the master processor compares the information on the piece 
of fruit with the classification criteria entered by the 
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operator. Once the piece of fruit has been classified, it 
moves along the outfeed conveyor 13 and upon reaching the 
outlet location corresponding to its classification, it is 
ejected from the conveyor. 
The Vision Processor 

The three main tasks of the vision processor are: 

1. Synchronisation of the video signal with the 
conveyor system; 

2. Digitising thd video signal; and 

3. Transforming the digital image into a useful form 
for analysis and passing onto the object 
processors . 

Figure 13 illustrates the function of the vision 
processor. 

The above functions will now be described in more 

detail . 

Synchronisati on of the Camera and the Convevoi- 

The conveyor speed of the fruit lanes varies 
according to the required fanait flow rate. The interval 
between subsequent photographs, however, is constant so it 
is necessary to coordinate the two different rates. The 
rate of mov^ient of the conveyor belt is indicated by the 
cup sjrnc signal. Each time a new piece of fruit enters the 
view of the camera, the cup sync signal goes high. Every 
time the camera takes a photograph, a camera sync signal 
goes high. It is necessary to determine the distance 
travelled by the conveyor during the time between the cup 
sync signal and the camera sync signal in order to 
determine the position of the fruit within the picture. 

Let the conveyor speed be in cups /second and one 
cup equal X metres, where for the purposes of this 
discussion, a cup is the length of conveyor able to handle 
an individual piece of fruit. Therefore, the conveyor 
speed is: 

V = X metres/second. 
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The elapsed time between cup sync and camera sync 
signals is T seconds, where the cup sync signal is a 
trigger signal relating to the arrival of each cup at a 
given location and the camera sync signal is a trigger for 
the next photograph. So the displacement of the image 
within the photograph is: 

VT metres . 

If the number of pixels in the images is P 
pixels /metre ^ then the displacement of the image is: 

) 

VTP pixels. 

This value VTP is used to effect the image (Figure 
4) to ensure that the object processors are sent data which 
has a fruit located centrally in the image section. 
Digitising the Image 

The signal from the camera is an analog signal 
comprising the red, blue and green components of the 
picture. For the purposes of image processing, it is 
necessary to convert the analog image into a digital image. 
The digital image comprises a matrix of pixels, each pixel 
. containing variable components of red, green and blue 

signals. The digitising of the pictures is performed by 
the Analog-to-Digitai <A/D) converters, shown in Figures 13 
and 14 . 

Transforming the Digital Image 

At this stage the picture taken by the camera and 
digitised by the vision processor, still contains up to 
eight pieces of fruit in the one image 201. It is 
necessary to convert this image into a useable form. This 
is done by converting the digitised image into eight 
individual image sections 202-209, each image section 
containing an image of one piece of fruit. In doing this. 
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it: is necessary to use the displacement of the image 
information calculated in step 1^ as described above. Once 
the jLmages are sectioned, they may be sent to the 
individual object processors for further processing of the 
images • 

The vision processor performs further 
transformations of the image in order to allow easy 
extraction of the relevant information. Before detailing 
the method of extracting relevant information, it is 
necessary to make a number of obsesrvations : 

1. The fruit is being sorted on the basis of its 
attractiveness to humans, hence the machine should ) 
respond by sorting it in a similar way. 

2. Humans have three colour senses, namely red, green 
and blue. People are very sensitive to green, less 
sensitive to red and very insensitive to blue. 

3. In judging colour, the eyes use little intensity 
information. Therefore, intensity may be discarded 
during colour processing. 

4. Blue component is the least significant in the 
colour of fruit. These characteristics are 
illustrated in Figure 15. 

The RGB colour space may be represented in three 
dimensions with red, green and blue axis. Exactly where a 
particular coloured point belongs in the space is ) 
determined by the contributions of these three colours. An 
example is illustrated in Figure 16 where this chosen 
colour activates the red sensor, and the green sensor. The 
blue sensor is not activated - this colour is called 
yellow. This chosen point has an intensity I, which is 
proportional to the distance between the point and the 
origin in the RGB space. 

This red-green-blue (RGB) interpretation, although 
not strictly correct in portraying the highly complex 
nature of colour perception, provides an extremely useful 
and practical model. 
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The functions of the transformations performed by 
the vision processor are to manipulate the raw RGB data 
into a form that can be mapped onto an easier to process 
two-dimensional surface and to reduce the effect of light 
intensity on sorting results . It is important to remember 
that light source intensity can vary considerably while a 
machine is operating. Not only can the light output vary 
due power supply variations and light source performance 
can deteriorate with age but also reflected light from 
surfaces adjacent to the source can become dirty and less 
reflective. 

A nximber of mathematical transformations on the RGB 
valves can largely eliminate the effect of brightness. 
This is done in the present embodiment with the use of a 
normalisation procedure^ described below. It should be 
noted that the vision processor sets the bright and dark 
levels by sampling reflected light from a red, green blue 
and black coloured strip adjacent the fruit conveyor. 

An example of one such transformer follows: 

In the cartesian axis colour space the intensity I 
is equal to the length of the line joining the origin and 
the point representing the colour. Consider point A in 
Figure 16. Any point along the line has an equal proportion 
of red/ green and blue so it is the same colour. Only 
intensity differs. 

The following normalisation transformations will 
project all points in the colour space onto the surface of 
a sphere with radius K (intensity): 

R-r 

Gt 


R K where RGB are red, green, blue 

I valves of given location in the 

G K colour space Rf, Gj, B-f are 

I the transformed valve for that 

B K location . 

I I = I ntensity 

"r^h- b"*" + g"^ 
K = Constant (=1) 
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This normalisation procedure substantially 
eliminates all intensity variations in the images^ thus 
leaving images when contains substantially only colour 
information which can be compared with other images and 
classification criteria , without the interference of 
unwanted information, such as distozrtions in the images 
caused by light variations. 

During the image grabbing procedure of the vision 
processor, it is necessary to capture as much of the 
surface area of the fruit as is possible, in order to make 
a correct judgement regarding classification. 

As has been explained above, in the preferred ) 
embodiment, up to four images are taken of each piece of 
fruit. Each image is taken at a regular interval. 
Between each image, the fruit move along the conveyor belt 
for a distance of M mm. The piece of fruit rolls along the 
conveyor system during this time, thus resulting in a 
different part of the surface of the fruit coming into the 
view of the camera. Between each picture the piece of 
fruit rolls approximately M mm along its circumference. 
Since a maximum of four pictures can be taken of each piece 
of fruit, the maximiim amount' of surface area of the fruit 
that will be photographed corresponds to about 411 mm of the 
circumference. If the piece of fruit has a circumference 
less than 4M mm, it will not be necessary to use all of the ) 
image information which is photographed, thus reducing the 
number of images to be considered by the object processors 
and resulting in faster classification of the fruit. 
Therefore, it is necessary to estimate the size of the 
fruit, in order to estimate the number of pictures which 
have to be considered during the classification process. 

Upon receiving the first image of a piece of fruit, 
the vision processor estimates the diameter, , of the 
piece of fruit. If the diameter of the fruit is less than 
4M, then it will be possible to photograph virtually all of 
the surface of the fruit. Let T = 4M . 
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If is less than T, then. the image processor 
takes a slice out of the image section of width M. This is 
shown in Figure 17A* This slice is sent to the object 
processor for further processing and use in the final 
classification procedure. When the vision processor 
receives the second image section, it again calculates the 
diameter of the object as It then estimates the 

diameter D of the piece of fruit by averaging D-^ and D2 . 
It is then necessary to calculate what percentage of the 
second image needs to be taken into account so as to ensure 
that there is no overlap with the previous image if the 
circvunference of the piece of fruit is less than 2M* The 
vision processor calculates the circximf erence of the piece 
of fruit as C = Dit. If C is less than 2M, then the vision 
processor will only take a slice of the image which is 
large enough to cover the whole of the circumference of the 
fruit/ as shown in Figure 17B. This slice is aM <a<l> 
wide. At this stage the vision processor would have 
captured images covering the whole of the surface area of 
the fruit, and can therefore stop processing further images 
of that piece of fruit. Again the picture slice is sent to 
the object processor for further processing. If C is 
greater than 2M, then the vision processor will take an 
entire slice of width M and proceed to take and process a 
third image. 

This process is continued until the entire surface 
of the piece of fruit has been photographed. 

If, when receiving the first image section, the 
vision processor determines the diameter to be greater than 
T, then it will not be possible to photograph the entire 
surface area of the piece of fruit. Consequently, the 
vision processor will capture the slice of width H, as well 
as slice A, as shown in Figure 17C. This will give the 
vision processor additional coverage of the surface ar a. 
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In the subsequent second and third images it only takes 
image slices of width but when it reaches the fourth 
image ^ it also takes the additional slice marked B in 
Figure 17D. In this way, it is possible to obtain images 
of virtually all of the surface area of a large piece of 
fruit • 

The Object Processors 

After the vision processor has divided, normalised, 
and manipulated the image sections, all the required image 
sections for each piece of fruit are sent to the individual 
object processors for detailed colour, defect and size 
analysis. Each object processor receives the image 
sections of a piece of fruit, and analyses the sections 
according to the requirements of the classification system. 

In the present embodiment, fruit are sorted 
according to their colour. The colour classification and 
selection system will now be described in more detail. 

Since the majority of fruit are of a red to orange 
to yellow colour such as oranges, lemons, apples and 
peaches, the blue component in the colour of the fruit can 
be substantially ignored. The present algorithm maps the 
transformed value Rc^, Grp, and Bij. onto the two dimensional 
space ^ with / values across the horizontal axis and B^ 
along the vertical axis. See Figure 18 • The system 
effectively plots the transformed colour values , , and 
Brp for each pixel associated with the surface area of a 
piece of fruit onto the two-dimensional plane and totals 
the numbers in each of the predefined colour areas. Since 
the majority of fruit are in the colour range red, orange, 
yellow, and green, most of the plotted points would be 
along the horizontal axis . The more uniform the colour of 
the fruit, the closer the plotted points will be together. 
Figure 19 illustrates the fruit being divided into four 
classification groups, comprising red, orange, yellow and 
green . 
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The master processor can be programmed to specify 
these different classes. As the pixels corresponding to 
the surface of the fruit are plotted on the plane, the 
master processor will select which class the fruit would 
fall in from the plot of its pixels. 

Once this classification process by the object 
processor is completed and the master processor has made 
the decision regarding which class the piece of fruit 
should fall in, the pieces of fruit travel along the 
outfeed conveyor system and will be ejected at the 
corresponding ends . 
Size Calculations 

Each of the four iiaage sections of a piece of fruit 
are broken down into the long thin rectangles covering the 
entire surface of the fruit. By adding the volumes of 
cylinders of rotation of the small rectangles, it is- 
possible to estimate the volume of the fruit using a 
Riemann Stim method. Using this method it is possible to 
then calculate the volume of the fruit as: 

n = a 
n = b 

constant 

first slice (first line of image) 
last slice (last line of image) 
diameter of slice (number of pixels 
across the slice) 


where : K = 
A = 
B « 
D = 


These calculations are carried out about two axes 
at right angles to each other, as shown in Figures 2 OA and 
20B. The lowest volume value of the two is selected and 
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the four lowest volumes (one for each of the four views) 
are averaged to produce a final estimate of the volume. 

The shape of the fruit can be deduced from a number ' 
of ways. Using all four image sections^ diameters of the 
piece of fruit are measured at 45 degree angles to each 
other. This produces sixteen estimates of the diameter of 
the fruit from the four views . 

In the case of substantially spherical fruit, all 
sixteen diameters are averaged to provide the mean 
diameter, see Figure 21. 

For flat fzruit, such as mandarines, the largest 
diameter for each view is selected and the average of these ) 
four values is representative of the largest circular cross 
section diameter of the fruit. 

For long fruits, such as lemons, the smallest 
diameter for each view is selected and the average of these 
four values is representative of the smallest circular 
cross sections. 

Additionally, the ratio of the largest to the 
smallest diameters from each view are averaged to produce a 
slendemess ratio. This ratio can be used to separate 
fruit of similar cross sectional diameter with largely 
varying lengths. For example, short almost round lemons 
can be packed separately from long slender lemons. 

All of this colour, defect and sizing information ) 
can be compared with operator preprogrammed data to ensure 
that fruit of particular size and grade are consistently 
ejected at specific outlets. 
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CLAIMS 

!• Apparatus for classifying and sorting objects 

comprising: 

a singulator to form the objects into a least one 

row; 

a conveyor that is arranged to pass at least one 
row of the objects underneath a camera, the camera being 
capable of capturing at least one image of the objects; 

means to rotate each object whilst passing 
underneath the camera; 

an image processing means for processing of the 
captured image, the image processing means including means 
for dividing the captured image into image sections, each 
image section containing a representation of only one of 
the objects, and the image processing means in use 
perfoirming further processing on each of the image sections 
s imultaneous ly ; and 

a control means for analysing output from the image 
processing means, classifying the objects according to 
predetermined criteria using the analysed output, and 
controlling ejection means to reject the objects at 
particular locations on the conveyor corresponding to the 
predetermined criteria . 

2. Apparatus according to claim 1, wherein the 
singulator comprises a horizontal conveying surface bounded 
on either side by inclined conveying surfaces operating at 
a slower speed than the horizontal conveyor surface. 

3. Apparatus according to either claim 1 or claim 2, 
wherein the conveyor comprises: 

a plurality of rollers spaced longitudinally in the 
direction of the conveyor to form at least: one row, 

means to drive the rollers along the conveyor, the 
singulator being positioned at one end of the conveyor with 
the inclined surfaces of the singulator extending alongside 
some of the rollers at the start of the conveyor. 
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4. Apparatus according to any one of the preceding 
claims, wherein the means to rotate each object comprises 
means to axially rotate the rollers, the object being 
located and supported between two adjacent rollers. 

5. Apparatus according to any one of the preceding 
claims, \«iierein ejector means are provided within the gap 
between adjacent rollers, means being provided to 
periodically actuate the ejector means to cause objects 
positioned between adjacent rollers to be flipped off the 
conveyor sideways of the rollers. 

6. Apparatus according to claim 5, wherein the ejector 
means comprises a plurality of downwardly extending 
abutment arms secured between adjacent rollers, 

means being positioned along the length of the conveyor 
periodically engage the arms to operate the ejector means. 

7. Apparatus according to claim 6, wherein a ramp 
element is positioned beneath the rollers adjacent the path 
of the abutment arms of the ejection means whereby, in use 
the ejector means on contact in the ramp are displaced 
either side of the ramp to operative or inoperative 
positions respectively. 

8. i^paratus according to any one of the preceding 
claims, wherein a pair of rollers are mounted axially 
spaced about a drive shaft, the drive shaft being clipped 
to at least one conveyor chain and the drive shaft 
including friction means in engagement with a fixed 
abutment to impart axial rotation to the shaft as the 
conveyor moves past the abutment. 

9. Apparatus according to claim 8, wherein each shaft 
is clipped onto the links of spaced apart drive chains, the 
rollers being positioned on the outside of the drive 
chains . 

10. Apparatus according to claim 9, wherein each 
ejector means is clipped onto the drive chain to extend 
between two adjacent rollers. 


wo 91/04803 


- 27 - 


PCT/AU90/00464 


11. Apparatus according to any one of the preceding 
claims^ wherein a plurality of outfeed conveyors are 
positioned along the sides of the conveyor, the outfeed 
conveyors being arranged to transport classified and sorted 
objects away from the conveyor as they are ejected from the 
conveyor . 

12. Apparatus according to claim 11, wherein adjacent 
each outfeed conveyor is positioned a cylindrical bmish 
that engages the objects as they are ejected from the 
conveyor and directs the objects onto the start of the 
outfeed conveyor. 

13. Apparatus according to any one of the preceding 
claims, wherein the camera captures the image in colour, 
such that the captured image represented by red, green and 
blue components. 

14. Apparatus according to any one of the preceding 
claims, wherein the image processing means includes means 
for digitising the captured image. 

15 . Apparatus according to any one of the preceding 
claims, wherein the image processing means includes means 
for normalising the captured image. 

16. Apparatus according to claim 15, wherein the means 
for normalising the captured Image includes a coloured 
strip for use as a reference in nosnaalising the captured 
image. 

17 . Apparatus according to any one of the preceding 
claims, wherein the image processing means includes means 
for determining the size, shape and/or colour of each 
object . 

18. Apparatus according to any one of the preceding 
claims, wherein the image processing means is comprised of 
a primary processing means having means for normalising and 
dividing the captured image, and a plurality of secondary 
processing means for processing the image sections 
simultaneously. 
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19. Apparatus according to claim 18, wherein the image 
processing means has eight secondary image processing means 
for processing the image sections simultaneously. 

20. Apparatus substantially as hereinbefore described 

with reference to and as illustrated in the accompanying t 
drawings . 

21. A method of classifying and sorting objects 
comprising the steps of: 

singulating the objects into at least one row; 
passing at least one row of the objects on a 
conveyor past a earner ia; 

rotating each object whilst passing underneath the j 

camera ; 

capturing at least one image of each object; 
digitising the captured image; 

dividing the captured image into a plurality of 
image sections, each image section containing a 
representation of only one object; 

analysing the plurality of image sections 
simultaneously to determine information on the colour, size 
and/or shape of each object; 

classifying the objects by comparing the determined 
information of each object with predetermined criteria; and 

selectively ejecting the objects from the conveyor 
at stations corresponding to the predietermined criteria. ) 

22. A method of classifying and sorting objects 
according to claim 21, wherein the captured image is 
normalised in order to substantially eliminate differences 
in intensity of the captured image. 

23. A method of classifying and sorting objects 
according to claim 22, wherein the captured image further 
contains a representation of a coloured strip, and the 
normalising of the captured image is made with reference to 
the coloured strip. 
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24 • A method of classifying anci sorting objects 

according to any one of claims 21 to 23, wherein the 
captured image is represented in red^ green and blue 
components . 

25. A method of classifying and sorting objects 
according to any one of claims 21 to 24, wherein the 
objects are sorted according to ranges of sizes, shape, 
and/or colour specified in the predetermined criteria. 

26. A method of classifying and sorting objects 
according to claim 25 when dependent on claim 24, wherein 
the green component is divided by the red component and 
plotted against the blue component, such that the ranges of 
colours are determined using the plotted components. 

27. A method of classifying and sorting objects 
according to any one of claims 21 to 26, wherein more than 
one images are captured such that all of the surface area 
of each object is represented. 

28. A method of classifying and sorting objects 
according to any one of claims 21 to 27, wherein 

an image is captured of eight of the objects to be 

sorted; 

the captured image is digitised; 

the captured image is divided into eight sections, 
each image section containing a representation of only one 
object; 

all eight image sections are analysed 
simultaneously to determine information on the colour, size 
and/or shape of each object; and 

classifying the eight objects by comparing the 
determined information on the colour, size and/or shape of 
each object with the predetermined criteria, 

such that the eight objects are sorted according to 
the predetermined criteria. 
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29 • A method of classifying and sorting objects 

according to any one of claims 21 to 28^ wherein 

first, second, third and fourth images of the eight 
objects are captured, and the objects are rotated in a time 
period between the capturing of each image, such that the 
images are views of different part of the surface area of 
each object; 

"the first, second third and fourth captured images 
are successively divided into- four sets of eight image 
sections, each image section containing one of the views of 
one of the objects; 

the eight image sections in each set are analysed 
simultaneously to determine information on the colour, size 
and shape of each object, the sets of image sections being 
processed successively; 

parts or the whole of one or more of the first, 
second, third and fourth views of each object are used to 
classify each object by comparing the determined 
information on the colour, size and/or shape of each object 
with predetermined criteria, 

such that the objects are sorted according to the 
predetermined criteria • 

30. A method of classifying and sorting objects 

substantially as hereinbefore described with reference to 
and as illustrated in the accompanying drawings. 


\ 
\ 


BNSDOCID: <WO_9104803A1J_> 


wo 91/04803 


PCT/AU90/00464 



BNSDOCJD: <WO_91 0480341 



BNSDOCIO: <WO_9104803A1J_> 




BNSDOCID: <WO_9104a03A1J^ 


PCT/AU90/00464 



wo 91/04803 


PCr/AU90/00464 



BNSDOCID: ^IWO_9104e03A1_L> 


^ WSTITUre SHEgf 


PCT/AU90/0a464 


7(14 



wo 91/04803 


PCr/AU90/00464 


8i\4- 




BNSOOCID: <WO 9104803A1 I > 


WO91/04803 



PCr/AU90/00464 



wo 91/04803 


PCT/AU90/00464 



BNSDOCID: <WO_9104803M J_? 


wo 91/04803 


PCr/AU90/00464 



wo 91/04803 PCT/AU90/00464 

Ml 14 



wo 91/04803 


PCT/AU90/00464 


\>)l4 


'1^ 


TO 


RED ORANGE YELLOW GREEN 


PURPLE CYAN 


BLUE BLUE BLUE 


) 



RED 
GROUP 

ORANGE 
GROUP 

YELLOW 
GROUP 

GREEN 
GROUP 





Bt 1 






M. 19. 


BN^PCID: ^yyp_ _9104803A1 1 > 


wo 91/04803 


PCT/AU90/00464 




a - 


DIAMETER 1 



DIAMETER 2 


DIAMETER 3 


DIAMETER U 


: 2L 


BNSDOCID- <WO_9104a03A1J^ 


INKERHATIONAI. SEARCH REPORT 

International ^Bcatlon No. FCT/ilT 90/00A64 


I, rf^<CTTnrCATr^ gTRTFrr matter m several classification syabols apply. 

indicate all) 

A 1 

According to International Patent Classification (IPC) or to both National Classification and 

IPC 

Int. Cl-^ B07C 5/342, 5/02 



TT WFITJR SRARCHED 

Mini BUM OocuMentation Searched 7 

Classification Systea | 

Classification Syabols 



IPC 

B07C 5/3A2 

Docuaentation Searched other than Hiniaun Oocuaentation 
to the Extent that such Docuaents are Included in the Fields Searched 

8 


i^: IFC as above 

HI. ixxassNTS catswEssD to be belevant 9 r 

Category* | Citation of Docuaentr ' ' with Indication, where appropriate, | 
1 of the relevant passages 12 | 

Relevant to 
Claia No 13 



X 

A 
A 

A 

A 

A 


(18.05.88) See itole docianent 

US.A, 4923066 (OffllR et al) 8 1990 (08.05.90) 
AT.A, 380814 (MASCfflNENFABRIK ANERITZ A.G.) 10 July 1986 (10.07.86) 
Patents Abstracts of Jsapan^ P-909, page 89. JP,A,1-105142 
(HITACHI PLANT & OONSUl 00. LTD) 21 April 1989 (21.04.89) 
Pateats Abstracts o£ Japan, F-909, pag^ 89, A,l*105141 
(HITACHI PLANT £NS & 0GN5IR CD. LTD) 21 i^sril 1989 (21.04.89) 
Patents Abstracts of Japan. P-909. page 87, JF,A,1*105137 
(HITACHI PLANT EN3 & (XNSTR CO. LTD) 21 isgoAl 1989 (21.04.89) 
Patents Abstracts of J^an, P-476. pagp 87. JP,A, 62-41905 
(TOSHIBA OORP) 28 February 1986 (28.02.86) 


24,25,27) 


• Special categories of cited docuaents: 10 

"A' docuiient defining the general state of the 
art which is not considered to be of 
particular relevance 

*£" earlier document but published on or 
after the international filing date 

'L* docuaent which aay throw doubts on priority 
claia<s) or which is cited to establish the 
publication date of another citation or 
other special reason <as specified) 

"O* docuaent referring to an oral disclosure, 
use, exhibition or other aeans 

"P* docuaent published prior to the 

international filing date but later than 
the priority date claiaed 


later docuaent published after the 
international filing date or priority date 
and not in conflict with the application but 
cited to understand the principle or theory 
underlying the invention 
docuaent of particular relevance; the 
claiaed invention cannot be considered novel 
or cannot be considered to Involve an 
inventive step 

docuaent of particular relevance; the 
claiaed invention cannot be considered to 
involve an inventive step when the docuaent 
is conbined with one or aore other such 
docuaents, such coabination being obvious to 
8 person skilled in the art, 
docuaent aeaber of the saae patent faaily 


IV. cekhficaticxi 


Date of the Actual Coapletion of the 

International Search 
j 2 January 1991 (02.01.91) 

I Znternationel Searching Authority 
I Australian F&tent OFHcr 


I Date of Hailing of this International 
I Search Report 

I Signature of Aut^ 


Ized Officer 


H.£. DIXON 


Fora PCT/ISA/210 <sec nd sheet) (January 1985) 


BNSOOCIO: <WO_9104«)3A1J_> 



ANNEX TO THE IlSflEraaMJCMftL SEftRCH HEPORT ON 
IMEEaiaTIOMftL ABELICftTIOM ND, K!T/Air907g0464 


This Annex lists tte kram "A" publication le^7el patent family msnibers 
relating to the patent documents cited in the above-^nentioned 
international search report. The Australian Patent Office is in no way 
liable for these particulars which are merely given for the purpose of 
information^ 


Patent Document 



Cited in Search 


Patent Family Members 

Report 


EP 267790 

US 4940536 



END OF ANNEX 


I 


BNSOOCID: <WO_9104a03A1J^ 


